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1. INTRODUCTION 



The United States Navy purchases millions of dollars worth of major 
caliber ammunition each year to supply its warships. These ships have 
missions ranging from air defense to gunfire support of amphibious forces on 
land. The rounds of ammunition purchased are each individual systems 
subject to various forms of failure, and are purchased by component. There 
currently exists no systematic method for determining the minimum 
acceptable reliability level for round components, reliability levels are 
currently established rather arbitrarily, but with some total system reliability 
in mind. 

The reliability of each of the components impacts the effectiveness of the 
weapon system and subsequently the effectiveness of the battle force. In order 
to procure and maintain ammunition which will provide adequate utility to 
Naval forces, there must be a clear understanding of this relationship of 
ammunition reliability and force effectiveness in various missions. 

Consider a single round of major caliber ammunition which is comprised 
of a small number of components, including fuses, primers, propellant 
igniters, propellant, propellant casing, case plug, projectile and explosive. 
Each of these major components is in turn an assembly of a number of 
subcomponents. A round may experience failures in each of these 
subcomponents. While some of the possible failures result in preventing an 
effective round from arriving at the target, other failures prevent the gun or 
gun cluster from being fired for some period after the failure. In relating 
reliability performance to battle force performances, attention must be paid to 
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time sensitive situations such as self-defense. The proper measure of 
effectiveness, and implied optimal reliability, must take into account the cost 
of extra reliability, the value of the target, and the impact of modes of failure 
on performance. 

The objective of this research is to establish reliability performance 
measures, and to specify minimum reliability goals for major caliber 
ammunition used in Naval Gunfire Support (NGFS) system. The reliability 
of ammunition and gun system must be expressed in terms of battle goals 
which include mission time, average casualty rate over mission lifetime, and 
the percentage of opponent's destructive power disabled at arbitrary time D. 
Then the decision maker has the ability to measure reliability performance in 
terms of the battle goals with which he is used to dealing. 

The approach we took in this research is described in Figure 1.1. We first 
developed a simple measure called ef which is a function of reliability 
measures of individual components of the ammunition and gun system. 
The (theoretical) ef measure represents the expected number of effective 
rounds per unit time. The round is considered to be effective if it gets out of 
the barrel, hits the target and detonates successfully. 

In order to determine reasonable reliability thresholds for ammunition, 
we must analyze the integrated system involving rounds, guns, targeting and 
scheduling. We compiled the data from various sources including Joint 
Munitions Effectiveness Manuals (JMEMS) and the OP-03 Material Readiness 
Data Base (MRDB) on the component reliability and gun reliability. This 
information is used to generate the NGFS scenario data file for simulation. 
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Figure 1.1. Flow Diagram of Reliability Goal Determination 
We developed a simulation model to find the relationship between ef 
values and the battle goals which we call standard performance measures. 
The model is designed to simulate the NGFS scenarios to mimic the spotting, 
communication and targeting. The simulation results show that the ef 
values are highly correlated with the standard performance measures. They 
explain variability in observations of the standard performance measures for 
different component reliability configurations. Thus the decision maker is 
able to determine the effectiveness of ammunition in terms of battle goals. 
He can find the minimum (or critical) ef value which is still mission 
effective. 
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This will support his decision making process on procurement and 
surveillance. In procurement decisions, he can specify reliability values for 
components and subcomponents which meet minimum requirements for ef 
in a most economical way. Ammunition surveillance is the practice of 
removing several rounds from a stockpiled lot of ammunition and testing 
the reliability of the rounds or of the components. He then can decide 
whether or not to perform some sort of rework on an ammunition lot to 
improve the reliability to bring it to mission effective status. Currently the 
acceptance criteria for pronouncing a stockpiled lot fleet-ready are ad hoc. Our 
research proposes replacing the current surveillance test criteria with those 
that will ensure that the lot tested exceeds minimum ef in efficiency. 

The results of our research can also be utilized to study the impact on 
marginal incremental improvements in component reliability for contractors 
or procurement agents. We provide an important set of tools useful in 
managing procurement and surveillance of ammunition. 

The structure of this report is as follows. In Section 2, we develop a 
measure of performance of the gun-round system, ef, which represents the 
number of effective rounds per unit time. A simulation model is described 
in Section 3, which is used to estimate the effectiveness of NGFS given the ef 
values. In Section 4, we explain how to run the simulation program, and the 
statistical analysis of simulation results. We also describe in some detail the 
variance reduction techniques applied to our simulation study, which results 
in substantial improvement of simulation output. Section 5 includes the 
conclusions and future research. 
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2. MEASURING PERFORMANCE OF MAJOR CALIBER AMMUNITION 



In this section we begin our exploration of the performance characteristics 
of the naval gunfire support system. In particular, we want to establish an 
appropriate performance measure to predict variations in the effectiveness of 
naval gunfire with respect to changes in round component reliabilities. 

The execution of an amphibious assault by combined naval and marine 
forces can be conceptually divided into two distinct phases. Phase one, which 
we shall call preemptive shelling, occurs prior to the landing of any marine 
forces. The objective of preemptive shelling is to reduce the capabilities of 
enemy shoreline defenses by bombarding defense positions. This 
bombardment is scheduled and directed by naval forces. The schedule is 
static; it does not change once the phase commences. Successful execution in 
the preemptive shelling phase will increase the probability of success in the 
second phase, which we call assault support. The transition to the assault 
support phase occurs when marine forces land and take their initial positions. 
The assault support phase is characterized by the continual receipt of requests 
for bombardment of specific shore defense positions. Calls for assault support 
shelling are made by the landed force and supported by spotters stationed near 
the targets. These calls for support shelling arrive intermittently and are 
serviced based on the priority of the call and the destructive power of the 
target. 
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2.1 MAJOR CALIBER AMMUNITION 



Consider a single round of major caliber ammunition. It is comprised of 
a small number of major components, including fuses, primers, propellant 
igniters, propellant, propellant casing, case plug, projectile, and explosive. 
Each of these major components is in turn an assembly of a number of 
subcomponents. 




Figure 2.1. 5-Inch, 54-CaIiber Propelling Charge and Projectile 
The key functional characteristic of this system is its strictly serial 
operation. For example, a typical detonating train consists of a chain of stages 
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intended to increase the fuze signal energy level to a degree sufficient to 
detonate the explosive payload. The output signal of the sensing equipment 
(radar signal, acoustic impulses, hydrostatic pressure, etc.) is sent to a primer. 
This device converts electrical or mechanical energy into explosive energy. 
Although this energy is minimal, it suffices to trigger an auxiliary detonator. 
Following this, the output power of the auxiliary detonator triggers a chain of 
insensitive explosives in the booster stage until detonation of the payload 
occurs. (See Weapons Systems Fundamentals, NAVORD OP 3000 Vol. 2, First 
Revision, for details.) Then for example, if the primer fails, no other 
component down the chain will have a chance to operate. It is this simple 
serial structure that greatly simplifies the analysis of the possible failure 
modes induced by the system, as it will be seen in the following subsection. 

2.2 FAILURES 

We will continue now with a structural analysis of the common modes 
of failure of the single gun round system. We will consider a round to be 
effective if it gets out of the barrel, hits the target and detonates successfully. 
Otherwise, if any of these conditions is not satisfied, the round will be labelled 
as non-effective. There are two operationally distinct failure classes, namely, 
interruptive failures and non-interruptive failures. We will assume that 
there is a total of N different types of failures with type Fi through type Fn-i 
corresponding to interruptive failures and type corresponding to a non- 
interruptive failure. 

An interruptive failure of type i {Fi, i = 1, ..., N-1) is caused by a 
malfunction of the round, the gun, or the crew, which causes the gun to stop 



2/22/91.11:38 AM.7 



7 



firing for some time T,. Examples of interruptive failures are firing pin 
failures, or failures in the propellant chain. An interruptive failure causes 
the system to stop firing for some time T,- in order to fix the problem, after 
which the system performs as before. 

A non-interruptive failure occurs when the round is successfully fired 
from a gun barrel but fails to cause effective damage to the target. Causes for 
this might be a faulty or incorrectly set variable time fuse, insufficient initial 
velocity, guidance errors, faulty detonation or even targeting errors caused by 
the crew. Although no effective damage is done to the target, a non- 
interruptive failure does not affect the gun in any other way, i.e., a non- 
interruptive failure does not affect the future firing rate of the gun. 

For the current analysis we have assumed that T,- , i = 1, ..., N are known 
and deterministic values. (We plan to extend this work to include randomly 
distributed repair times). We will also normalize these values according to 
the firing rate of the gun, i.e., if the gun is firing at 18 rounds/minute, we 
then measure T,- in units of 1/18 minutes. In this manner the down times 
caused by each particular failure will be weighted proportionally to the firing 
rate of the gun, so that a 10-minute interruption is considered worse for a gun 
shooting at a high firing rate than a gun shooting at a lower rate. This also 
implies that an effective round and a non-interruptive failure each take one 
unit of time. 

Consider again a single round of a major caliber ammunition. The linear 
nature of this system allows us to model the system so that the lower levels of 
failure take precedence over the higher levels. Then, the failure (conditional) 
probabilities that can be estimated by observation are given by 
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xi = P[F\ occurs] 



Xi = P[Fi occurs I no Fy occurred for j < i], 

for /■ = 2, N. Hence, the probabilities of the different failure types may be 
computed as 

qi = P[Fi occurs] 

qi = P[Fi occurs and no Fy occurs for j < /]. 

By the way we have defined the observed conditional probabilities (x,'s) the 
failure types are independent and therefore 

= (2-1) 

/<*■ 

for 1 = 1 ,.... N. Thus, the qi values will allow us to compute the reliability 
measure for the single gun-round system. 

2.3 A MEASURE OF PERFORMANCE OF THE GUN-ROUND SYSTEM 

We now present a measure of performance of the gun-round system 
based on the time required to deliver an effective round on the target. Recall 
that a round is considered effective upon successful detonation on the target. 
Let the random variable Tp represent such time measured in firing epochs, 
then 

N 

( 2 - 2 ) 

1=1 
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where Pi is the number of failures of type i occurring between effectively 
delivered rounds, i.e., the Pi's are geometric random variables with parameter 
Pi = 1 - qi, for i = 1, N. Thus, the first two moments of Tp are given by 

N 

E[Tp]=X^(l-P.)/Pi + l' (2-3) 

J=1 

N 

var[Tp] = (2-4) 

1=1 

There is a very important relationship between these two moments that 
serves to consolidate our objectives, namely, monotonicity. As it will be seen 
in the following theorem, the expected value and the variance of the random 
variable Tp increase and decrease together. In evaluating the performance of 
a given set of components with associated xi values, low variance of Tp is 
desirable as well as low expected value. This theorem guarantees that 
components giving lowest expected time between effective rounds also give 
the lowest variance. Thus, we are interested in decreased values of both E[Tp] 
and var[Tp], and these goals do not conflict in any way. 



THEOREM 2.1. Given two alternatives p and p in [0, 1]^, if E^Tp^ < e[T'p|, then 
var[Tp] < var[Tp]. 

PROOF. We will show that for two points p and p suitably close to one 
another, that VEj^Tpj (p-p) and Vvarj^Tpj (p-p) have the same sign. Thus, 
descent directions for E[Pp] are also descent directions for var Tp . Consider 
the product 
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VE[Tp] (p-pf Vvar(Tp) (p-p) 
= (p-pfVE[Tp]Vvar{Tpf (p-p) 



-T'vPi 



= (P-Pf 



[(p,-2)Tf/pl- 



-Tn / Pn 



^(Pn-^)Tn /Pn]{p-p) 



= {p-pf 



(2-p,)T-,3/pf 
(2-P2)72 /P 2 P 1 

(2-P3 )t| /pIp^ 



(2-Pi)7^72 /P?P2 
(2-P2)72 /p| 
{2-p3)TiT2 /pIpI 



(2 - Pi )7^T3 /p^pl ■■■ (2 - p, )Ti 2 t^, / pi^p^ 
(2 - P2 )'^ 2^3 / P2P3 (2 - P2 )”^2^Tn / pIpn 
{2-p3)Ti/pl i 






{‘^-Pn)'^n /PnPi 



(2 -PnKn ! Pn 



T 

= {p-p) Mp-p) 



(2.5) 



which is a quadratic form in the matrix A. It is simple to determine that A is 
positive semi-definite. This tells us that 



VE 






(p-p) 



T 

Vvar(Tp) (p-p) 



>0 



( 2 . 6 ) 



for any vector (p-p). Hence, we may conclude that in a suitably small 
neighborhood of any point p, descent directions for E[Tpj are also descent 
directions for varj^T^j. □ 

Backed by the results of the above theorem we can now define a measure 
of performance of the gun-round system as: 

ef = 11 = round efficiency. (2.7) 



This quantity represents the number of effective rounds per unit time (again 
measured in firing epochs). As will be seen in the following section the 
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simulation results confirm that ef represents a good measure of performance 
of the naval gunfire support system, highly correlated with both the 
destruction time of the target and the destructive power left by time f. 

Example 1 

Eight failure types known to cause a preponderance of the failures 
experienced by a naval gun are given below in Table 2.1 along with their 
interruption (repair) times. These times were derived from studying the 
figure "Hot Gun Misfire Procedures and Safety Time Schedule for 5-inch, 54- 
caliber Guns Mark 45 Mods 0 and 1". Failure type descriptions are given in 
Table 2.2, with some symptom information, remedy, and the interruption 
time for each. When two numbers are provided they are for hot and cold gun 
situations. Repair of these types of failures on a hot gun require that the gun 
be cooled for 30 minutes before repair can begin. This assumes that an 
automatic internal cooling system can be employed. The wait is two hours 
with no cooling system. Finally we evaluate the efficiency of a round under 
these conditions. 



TABLE 2.1 



Failure Type 


Description 


Data Status 


1 


Loader Microswitch Fails 


E 


2 


Firing Circuit Fails 


E 


3 


Breachblock Closed 


E 


4 


Firing Pin Fails 


E 


5 


Propellant Primer Fails 


D 


6 


Propellant Impurity 


D 


7 


Fuse Fails or is Missed 


D 


8 


Detonation Train Fails 


D 
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The qualifier data status is E if the probability of failure must be estimated 
from data provided from a gun failure data base, while those with data status 
D are failure types whose probabilities are variable, open to decision. 



TABLE 2.2 



Fi 


Description/Symptom 


Remedy 


Ti 


Pi 


1 


Round jammed, loader stops 


Mechanically 

unjam 


25/55 


0.05 


2,3 


Electronic failure in system 


Isolate and 

replace 

component 


1.5 


0.1 


4 


Electronics work but no fires 
occur 


e.g.. Replace pin 


21/51 


0.02 


5 


Propellant will not fire though 
mechanical firing sequence 
occurs 


Eject propellant 
casing and fire 
clearing round 


10 


0.03 


6,7,8 


Non-interruptive failure 


Shoot again 


1 /firing 
rate 


0.01 



Based on the above data and the equations provided in the previous 
subsection we evaluated the efficiency of the round for a gun shooting at 20 
rounds/minute and obtained ef = 0.16, i.e., 16% of each round is effective. In 
other words, if we assume that 10 rounds are necessary to destroy the target, 
then given ef = 0.16, we will need an average of 62.5 firing epochs to deliver 
10 effective rounds. 

2.4 PURPOSE OF e/ 

As it will be seen in the following section, the simulation model (and the 
associated computer program) will allow us to determine the appropriate 
value of ef in order to meet battle goals with respect to the time required to 
destroy all targets, the casualty rate, destructive power left by time D, etc. The 
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selected number for ef, say ef*, will thus represent a global measure to be 
satisfied by all types of ammunition and guns in the system, ensuring in this 
way that the battle objectives will be met. 

It should be noted that many individual components are either common 
to different guns or to different ammunitions, e.g., two different models of 
the same type of gun may share the same firing circuit. Therefore, the 
decision as to what the minimum reliability requirements are for all the 
components in the system should be taken globally, i.e., the result of a model 
that includes all guns and all ammunitions simultaneously. In other words, 
each different round of ammunition and each gun in the system will have to 
satisfy an equation of the type E[Tp] <11 ef*. 

A system like the one described above possesses an infinite number of 
solutions. It is possible, in order to simplify the selection of a particular 
solution, to include an objective function, e.g., minimum procurement cost. 
This last step is dependent upon knowledge of the different costs associated 
with buying components of different reliabilities, information that we 
understand may not be readily available to the decision maker. 
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3. SIMULATION MODEL 



We have developed a simulation model to estimate the relationship 
between ef and other standard performance measures which include mission 
duration, average casualty rate over mission lifetime, and the percentage of 
opponent destructive power disabled at arbitrary time D. The model took the 
following aspects into account; 

i) gun reliability 

ii) spotting, 

iii) targets and target hardness, 

iv) navigation error, 

v) guidance error, 

vi) fluctuation in initial velocity 

vii) ships and guns and dependence, and 

viii) all possible JMEMS data. 

The goal of the simulation model was not to simulate a particular known 
engagement, but to give decision makers a feel for the impact of varying ef 
values in terms they are used to considering (standard performance 
measures). The simulation results show that ef explains nearly all variability 
in observations of the percentage of targets destroyed at arbitrary time D, the 
time required to destroy all the targets, and the casualty rate. 

The simulation will also provide the decision maker with the statistical 
properties of these measures. Thus the decision maker has the ability to 
measure reliability performance and compare previously incomparable 
component configurations. 
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The decisions he will make involve procurement and surveillance. In 
procurement decisions, he will specify reliability values which meet 
minimum requirements for ef, and which are most economical. In 
surveillance, disassembled rounds will be tested piece by piece. If the round 
still is within the minimum ef standard, it is still mission effective. If not, 
the rework required to bring it to mission effective status can be prescribed in 
a cost-effective way. Thus the critical value of ef must be established. 

3.1 THE MODEL 

The model is designed to mimic the spotting, communication, targeting, 
and destruction aspects of the NGFS system. The system consists of several 
ships, each equipped with one or more naval guns. These guns are tasked to 
prosecute and destroy a prespecified set of targets by a specified time. 
Interfering with the gunfire are inaccuracies in the fall of shot, inaccuracies in 
the ship's internal navigation system, gun system interruptions and failures, 
and ammunition failures. We now describe the simulated processes in more 
detail. 

3.1.1 The Target List 

The specified targets reside on a target list which gives each target's type, 
the lethality points for the targets, and each target's field of view. The target 
type is referenced to a target type list which gives detailed information on the 
method of engagement for the target type. An example of a target type may be 
a TEL of an antiaircraft gun, while a target might be a particular TEL in a 
particular location. 
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Point values are assigned to each of the targets to control the manner in 
which they are engaged. Generally speaking, targets are engaged in a highest- 
points-first regime, with exceptions made for partially destroyed targets. 

When the NGFS scenario begins, the targets with the highest point 
values are taken from the target list and engaged, one target per gun. Each 
target is prosecuted until it is destroyed, or until the gun engaging it suffers a 
severe casualty. If the target is disengaged and not destroyed, it is returned to 
the target list in its current state of damage. The scenario is not considered 
complete until all targets on the list have been destroyed. 

3.1.2 Spotting 

Each target is first engaged by using a bracketing method for spotting the 
rounds. The brackets are established as shown in Figure 3.1. Two points are 
established behind (Pl for Plong) in front of (Ps for Pshort) the target. 
Around each point, a box is established. The evolution starts as the ship 
attempts to place a round in Box L, Xl is an acceptable long spotting round. 
Then, the ship attempts to place a round in the short box, say at Xs. The 
distance from Pl or Ps to the target, and the width and length of the spotting 
boxes are specified for each target type. Note that if the distance from P$ to the 
target is specified as 0.0, the gun performs single-shot spotting on the target. 

The preceding description fits the actions of a ship engaging its first target. 
If, however, the ship has just destroyed an adjacent target, this bracket 
procedure may be overly cautious and time consuming. Hence, if the ship 
immediately engages a target within the same field of view as the target just 
destroyed, the ship performs single-shot spotting instead of the bracketing 
procedure. 
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Figure 3.1. Target Spotting 

3.1.3 Firing for Effect (FFE) 

Once adequate spotting has been performed, the ship will fire for effect on 
the target. This involves firing at a position exactly between the two spotting 
locations, as at the last single-shot spotting location. The gun fires a salvo the 
size of which is specified by the target type, each round following the previous 
one by a delay equal to the specified cycle time of the gun. 

3.1.4 Destruction 

The combination of the impact point, the round type, and the target type 
allows us to specify the amount of damage done to the target. Each round- 
target type gives us an effective casualty radius (ECR) within which some 
damage to the target can be expected. Each target type has a specified number 
of lifepoints, and a number of points it loses if the round lands within the 
ECR and fuses successfully called hitpoints. We further downgrade the 
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effectiveness of the round by a probability of damage, which is conditioned on 
the round landing within the ECR and fusing. Between the hitpoints, 
lifepoints, and this conditional probability, the analyst has significant 
modeling flexibility. When a target's lifepoints are no longer positive, it is 
destroyed. 

3.1.5 Battle Damage Assessment 

After an effect salvo is fired, there is a delay before the disposition of the 
engaged target is known to the ship. Contingent upon the condition of the 
target, the ship may do one of three things. If the target is unharmed, the ship 
attempts to respot the target; if the target is damaged but not destroyed, the 
ship fires another salvo for effect; if the target is destroyed, the ship engages a 
new target. 

3.1.6 Navigation Error and Registration Rounds 

Prior to attempting to spot its first target, each ship experiences some error 
in its navigation solution as well as some gun-system bias. These two effects 
conspire to make the gun system less accurate. It is a common practice to fire 
several registration rounds at a point outside the target field, and to track the 
landing point of these targets using the ship's radar. From this data we form 
an estimate of the bias carried by the ship's navigation system so that future 
firings are more accurate. 

3.1.7 Found Aimpoints and Spotting Inaccuracies 

When the ship fires a round, it does so at a specified aimpoint (xA,yA) 
given in (x,y) coordinates. The round, due to several exogenous conditions, 
lands at some nearby point (xi,yi). In our model, we assume that the 
difference between (xA,yA) and (xi,yi) is a bivariate normal random variable. 
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With mean (0, 0) and variance which is specified in the data set. Each gun is 
allowed a different variance, and each round type may magnify this 
inaccuracy as appropriate. 

When a spotter observes the impact point relative to the target, he 
advises the ship to adjust the aimpoint by the observed error, as opposed to 
the aimpoint's inaccuracy. See Figure 3.2. 




Figure 3.2. Aimpoint Adjustment 

This process continues until the impact point satisfies the spotting criteria. 

3.2 GUN AND ROUND FAILURES 

Of primary interest in this research is the effect that gun system and 
ammunition failures have upon the overall performance of the NGFS 
system. As discussed above, the system experiences failures which are of two 
distinct types, these being interruptive and noninterruptive failures. Each 
failure is modeled to occur with a given probability of each firing of the gun, 
and each failure is assumed to be independent of all previous firings. 
Furthermore, the observation of failures is hierarchical, so that the 
occurrence of a failure masks occurrences of failures of components further 
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down the detonation train. Upon recovery from a failure, the system is 
assumed good-as-new. 

3.2.1 Noninterruptive failures 

Modeling noninterruptive failures is quite straightforward. If one of 
these failures occurs during the spotting process, the round is assumed to be 
unobserved by the spotter. Hence, the spotter will require that an additional 
spotting round be fired at the current spotting aimpoint. During the FFE 
phase of the mission, the round is simply ineligible to inflict damage upon 
the target. 

3.2.2 Interruptive failures 

The duration of an interruptive failure changes depending on whether 
the gun is hot or not. Upon an interruptive failure during either phase of a 
mission, there is a possibility that the gun should be taken off-line to be 
repaired, and the engaged target should be made available to other guns. This 
target release evolution occurs when the gun is expected to be down for more 
than TOO LONG time units. The target becomes reengageable by another gun 
after RELEASE TIME units. The values of TOO LONG and RELEASE TIME 
are specified in the scenario. 

If the failure is not severe enough to release the target, the actions of the 
system depend on the mission phase at the time of failure. If spotting is 
interrupted by a failure, the spotting resumes by firing at the original spotting 
location. If the gun is firing for effect, the resumption of firing involves 
shooting the entire salvo over again. However, when FFE is interrupted it is 
possible that the target has already been destroyed. Thus, BDA is done upon 
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every interruption of an FFE. An interruption of an FFE could, in this 
manner, result in reassigning the failed gun to a new target upon recovery. 

3.3 DATA FILE PRODUCTION 

The generation of data for the model falls into three general areas. First is 
the collection and evaluation of scenarios. Second is the translation of 
known targets to JMEMS targets and the evaluation of munitions effects. 
Third is the evaluation of gun system reliability. All of this data is destined 
for the scenario data file of the simulation AMMO. 

3.3.1 Collection and Evaluation of Scenarios 

There were three scenarios considered for use in the study, based on 
amphibious actions in the Middle East, northern Europe, and the northern 
Pacific. The Mideast scenario is a helo-borne landing and therefore does not 
meet the requirements of the study. The Northern Europe scenario is an 
unopposed landing and, thus, does not meet the requirements of this study 
either. The third scenario was a northern Pacific scenario with many targets 
designated to be assigned to prelaunch strikes by naval air and naval gun 
assets. Specifically 18 targets were evaluated as being assigned to NGFS. The 
NGFS time window for these targets is the 48 hours preceding the scheduled 
landing time. We used this scenario as the basis of our model runs. 

Specific detail about the targets other than location, elevation and general 
nomenclature was not available, therefore assumptions had to be made about 
the targets. It was assumed, for example, that AA battalion headquarters 
would be fortified in such a manner as to present the damage profile of an 
armored personnel carrier. It was assumed that each surface-to-air missile 
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battery had eight surface-to-air missiles. It was also assumed that an aviation 
regiment would be comprised of eight aircraft. With regard to the 
navigational error, it was assumed that beacons were set in place on the beach 
for geographical reference and aid in navigation. Ships were assumed to be 
able to maneuver, gain, and maintain, a range of 12,000 yards to each target as 
it was taken under fire. 

3.3.2 Target Translation 

The scenario generated the following target type list: 

EW SITE 
SW SITE 
GCI SITE 

AA BATTALION HEADQUARTERS 
AA BATTERY 
HELO SQUADRON 
SAM BATTERY 

SAM BATTALION HEADQUARTERS 
AVIATION FIGHTER SQUADRON 

The JMEMS target list considered was as follows: 

140MM ROCKET AND LAUNCHER 

FROG 4 ROCKET 

152MM FIELD GUN 

ZIL 157 TRUCK 

SEPAL CRUISE MISSILE 

CROSS SLOT RADAR 

FROG 7B ROCKET POE^T AND AREA TARGETS 
BMP ARMORED INFANTRY VEHICLE 
PERSONNEL TARGETS 

For the target translation, EW, GCI, and SW sites were considered to be 
predominantly antenna arrays and electronics equipment, therefore they 
translated to CROSS SLOT radar for obtaining input data for the simulation 
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from the JMEMS publications. HQs translated to hardened targets with a 
larger target radius than most hard targets. AA battery translated to a 152MM 
field gun. Helo and aviation squadrons translated to sepal cruise missiles. 
SAM battery translated to FROG 4 rocket. 

From this translation we were able to generate the target type list in the 
data file described in Section 4. 

3.3.3 Gun System Reliability 

The 5-inch Gun Mount has 53 reliability blocks which are all in series. A 
reliability block diagram is a model of a system or operational mode, broken 
down to a level in which all components in each block are either all 
energized or all nonenergized at any point in time (NAVSEAINST 3500. lA 
Advance Draft). Reliability block diagrams (RBD) are used by the Fleet 
Analysis Center to determine reliability measurements of Navy ship systems. 
The RBDs for a particular system were developed by the engineering and 
maintenance support activities at the direction of the cognizant system 
program manager. 

The OPNAV 4798/2K is the primary source for the Material Readiness 
Data Base (MRDB). In addition the MRDB is augmented with casualty report 
messages, employment tapes, and other supplementary data sources. Data 
collected from the start of the data base in January 1984 until August 1986 
includes information from the NAVSEA Non-expendable Shipboard 
Equipment Status Logs, form 4855/2, which are no longer required. Data 
from 1984 to 1986 was used because it includes the equipment status log 
information. It is a source of continuous real time observation and actual 
clock readings. The mean time between failures (MTBF) and the mean time 
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to repair (MTTR) were obtained for each of the 53 RBD's from the MRDB. 
The RBD MTBFs and MTTRs were used instead of the system reliability to 
avoid the data base scenario adjustments of system reliability that are made 
using a demand factor for each of the RBDs based on the percentage of time it 
is energized during a typical mission life normalized by a ratio of energized 
time to real clock time, aggregated over all units reporting. We wanted to use 
as pure a number as possible and therefore modeled the individual 53 RBDs 
in the simulation and used the unadjusted data from the 1984 to 1986 time 
frame. 

3.3.4 Data Collection Postmortem 

There is a mismatch between those targets that we are describing in our 
scenarios and the information about the specific effect of our ammunition on 
targets that we were able to develop at this time. We are in the process of 
getting the effects data needed from the JMEM Munition Effects Working 
Group. Our scenarios should be generated with the same level of granularity 
or scale as the JMEMS publications, so that translation, if required, can be 
done in reasonable fashion. In terms of our own systems, there needs to be a 
methodical way to collect information in order to support reliability 
calculations, and to do so in such a way as to isolate those periods of time 
when the systems are specifically in a mission state. It does no good for us to 
evaluate our systems at 90 percent reliability, lets say, according to the data 
base when the system does not work 50 percent of the time while it is on the 
firing line. System reliability for a mission may not be accurate for the mark 
45 mod 1 system because the collected data is based on clock time, not mission 
time. 



2/22/91.11:38 AM.25 



25 



Because of these factors, it was deemed valuable that NGFS be observed so 
that some idea of validation of the model could be obtained. The observation 
of one ship does not suffice to validate the simulation model, but it did not 
invalidate the model. It was encouraging to see that the rate of fire of the ship 
in actual clock time was similar to that predicted by the model. It was also 
encouraging to see the gun perform as well as it did as the ship's crew became 
acclimated to the environment of NGFS. 

4.0 RUNNING THE SIMULATION 

In this section, we describe the data files required to operate the 
simulation model, and the output attainable. 

4.1 DATAFILES 

There are two data files required to run the simulation AMMO. These 
files are known as the scenario and run control files. The scenario file, an 
example of which is shown in Figure 4.1, contains three distinct sections: 
timing information, target description information, and ship-gun-round 
system description information. Finally, there is a list of seeds used in the 
simulation. 

All of the data collected from the FLTTAC, JMEMS, and other sources is 
destined for the scenario data file. The scenario supplied by NWSC was used 
to choose the targets and target types, and to specify the numbers of ships and 
guns. In addition, the data on the timing and accuracy of the gun system was 
provided by rough estimates from data collected aboard the USS Ainsworth. 
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TIMING INFORMATION 

5.5 bdatim, the time (in min.) it takes to do BDA 

3.5 spttim, the time between spotting round 

2 numgun, the number of guns in the scenario 

1 numshp, the number of ships in the scenario 

2 numshl, the number of shell types used in the scenario 
4 numtgt, the number of targets 

2 numtty, the number of target types 
100. dtime, the time of the amphibious assault 

1 dswtch, 1 means the simulation stops at dtime, 0 means destroy all targets 
4 regmd, number of registration rounds each ship shoots at the beginning of each 
scenario 



TARGETS — id tgttyp points fov (one for each target) 

. one 1 , 100 1 

two 1 99 2 

three 2 45 1 

four 2 56 2 

TARGET TYPES — id, shltyp, lifpts, hitpts, effect, ecr, deflct, jmemsp, ecrlng, pcorct, blng, 
bwd 

type 1 1 10 3 5 50 20 0.90 1 1.0 1.5 1.0 

type 2 1 10 3 5 50 20 0.90 1 1.0 1.5 1.0 

SHIPS — ship nav error sigshp (., 1) sigshpC., 2) 

100.0 100.0 standard errors, X and y, of the ship nav system 

GUNS — ship, gnumf, hotmd, siggunf., 1), siggunf., 2), cycle (one per gun) 
gfailp, gfailt(cold, hot) (one per failure mode of the gun) 



1 


2 


10 


20.0 


60.0 


0.05 


0.05 


1.0 


2.0 








0.10 


0.0 


0.0 








1 


2 


10 


20.0 


60.0 


0.05 


0.05 


1.0 


2.0 








0.10 


0.0 


0.0 









SHELLS — snumf, sigshl(., 1), sigshK., 2) (one per shell type) 
sfailp, sfailt(hot and cold) (one per shell failure mode) 
4 1.0 20.0 

0.5 5.0 10.0 

0.01 0.0 0.0 

0.03 3.0 3.0 

0.002 4.0 14.0 

3 1.0 0.0 

0.01 0.0 0.0 

0.03 3.0 3.0 

0.002 4.0 14.0 



Figxxre 4.1. Scenario Data 
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The blocks of data pertaining to the guns and shells may be nonintuitive 
in their design. The first line for each gun description gives, in order, the 
ship on which the gun is mounted, the number of failure types for the gun, 
the number of rounds fired before the gun is considered hot, the standard 
errors of the dispersion of shot for the gun (x and y), and the time between 
rounds fired for effect for the gun. The next several lines of data are the 
probability of occurrence and the recovery times (cold and hot) for each 
failure type. 

The blocks of data pertaining to the shell types are structured similarly. 
The first line gives the number of failures for the shell type, followed by the 
additional dispersion caused by the round. The following lines give the 
probability of occurrence, and the recovery time (cold and hot) for each failure 
type. 

A second file, the run control data file, contains some simple information 
used to limit the runs of the simulation and to implement our experimental 
design. An example of the run control file is given in Figure 4.2. The run 
control file specifies the maximum duration of any single simulation run, 
MAXTIME, the level of accuracy required of each of the performance 
measures, PCT, and the maximum number of replications a given 
experiment is allowed to run, MAXREPS. For example, if MAXREPS is 200 

and PCT is 0.05, then the simulation experiment will stop when 

1) the confidence interval half width is less than 5% of the mean for each 
performance measure, or 

2) 200 replications have been completed. 

Lines 4-6 of the file concern the "scrunching" of ECR values. As has been 
mentioned in the discussion of JMEMS data, it was often seen that a given 
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round-target pair would have an effective casualty radius of 850 meters and a 
probability of destruction of 0.01. We found this unreasonable and devised a 
crude way to make the probability of destruction increase by insisting on more 
accuracy in round delivery. 



200 




MAXREP 


35,000.00 


T 


0.15 

F 

40.0 

1 


0.000 


PCT 

T TO MODIFY THE DISKS, F TO SKIP THAT PORTION 
R (= ADAPTIVE RADIUS WE USE TO SCRUNCH ECRS) 
0.0005 


1 


0.00 


0.004 


1 


0.00 


0.01 


1 


0.00 


0.004 


1 


0.000 


0.002 



Figme 4.2. Run Control File 

We made the assertion that, within limits, ECR x Pr[destruct] = C for 
some constant C corresponding to the firepower of the round against the 
target. The aforementioned limits are established in lines 4 and 5 of the data. 
R gives the minimum ECR value for any round-target combination, while 
PMAX gives the maximum Pr[destruction]. The program, when instructed, 
will attempt to concentrate firepower until one of these bounds becomes 
tight. Experimentally, we have found that the simulation predicted highest 
gun efficiency when R was equal to the round dispersion standard error given 
in the scenario data file. 

The last several lines give the experimental design, which is of the 
complete factorial type. We are interested in the behavior of each of our 
performance measures when the probabilities of failure of each of the round 
components are varied. By convention, we vary the probability of failure for 
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round type 1 in the experiment. If the scenario involves several round types, 
the user should manipulate the scenario data file so that round type 1 is the 
round of primary interest for the run. 

The design is given by the number of levels, the lower limit, and the 
upper limit of the probability of failure for each failure type for round type 1. 
Thus, if the line corresponding to failure type 1 in the run control file is given 
as 

3 0.01 0.03, 

then type 1 failure probabilities will vary over the set (0.01, 0.02, 0.03}. If the 
number of levels for a failure type is given as 1, the probability of failure 
reverts to that found in the scenario data file. 

4.2 Model Responses 

When the model runs, the user's screen is notified of the number of 
experiments that will be performed in the run, the number completed thus 
far, and the number of replications finished for the current experiment. 
Upon completion of each experiment, the user is notified of the number of 
replications of the experiment which were censored because the targets were 
not all destroyed before MAXTIME. The number of censored runs should be 
small (ideally 0!) for each experiment. If this number becomes a significant 
portion of the sample size for the experiment, the values of the scenario, 
failure, or MAXTIME, should be reconsidered. 

When the simulation does run successfully, the output is given in the 
form shown in Figure 4.3. Estimates of the primary measures of 
performance, the mean scenario completion time, the time average of the 
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KEEPON = F 0 OF lOREPS CEIISORED 


THE 


NUMBER OF REPLICATIONS REQUIRED WAS 




10 




THE AVERAGE MISSION LIFETIME = 




563.5317 




VARIANCE OF THE MEAN = 




1146.4990 




STANDARD DEVIATION OF THE MEAN = 




33.8600 




THE TIME AVG OF THE TARGET VALUE = 




77.2558 




VARIANCE OF THE MEAN = 




23.3675 




STANDARD DEVIATION OF THE MEAN = 




4.8340 


ROUNDS PER MINUTE BY GUN 






1 


1.289 WITH STD ERROR 


.146 




2 


1.612 WITH STD ERROR 


. 101 




3 


1.326 WITH STD ERROR 


. 148 




4 


1.387 WITH STD ERROR 


.124 




5 


1.338 WITH STD ERROR 


.120 




6 


1.722 WITH STD ERROR 


.149 




THE 


GLOBAL AVERAGE 








1.446 WITH STD ERROR 


. 059 




THE AVG % OF TARGETS DEAD AT DTIME = 




1. 0000 




VARIANCE OF THE MEAN = 




.0000 




STANDARD DEVIATION OF THE MEAN = 




. 0000 


THE AVG SURVIVING TGT VAL AT DTIME = 




.0000 




VARIANCE OF THE MEAN = 




.0000 




STANDARD DEVIATION OF THE MEAN = 




. 0000 


GUN 


1 THE AVG ROUNDS PER MISSION = 


456.8999 




VARIANCE OF THE MEAN = 




2833 . 6540 




STANDARD DEVIATION OF THE MEAN = 




53.2321 


GUN 


2 THE AVG ROUNDS PER MISSION = 


473 . 2000 




VARIANCE OF THE MEAN = 




3422.3510 




STANDARD DEVIATION OF THE MEAN = 




58.5009 


GUN 


3 THE AVG ROUNDS PER MISSION = 


531.3000 




VARIANCE OF THE MEAN = 




3943.9120 




STANDARD DEVIATION OF THE MEAN = 




62.8006 


GUN 


4 THE AVG ROUNDS PER MISSION = 


440.3000 




VARIANCE OF THE MEAN = 




3836.4900 




STANDARD DEVIATION OF THE MEAN = 




61.9394 


GUN 


5 THE AVG ROUNDS PER MISSION = 


402 . 5000 




VARIANCE OF THE MEAN = 




4145.2280 




STANDARD DEVIATION OF THE MEAN = 




64.3834 


GUN 


6 THE AVG ROUNDS PER MISSION = 


487.2000 




VARIANCE OF THE MEAN = 




5527.7960 




STANDARD DEVIATION OF THE MEAN = 




74 . 3491 



Figure 4.3. Example Output File 
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surviving target value, and the target value surviving at DTIME are 
prominently displayed. In addition, we report several sets of statistics 
concerning the experiences of each gun used. This report is generated by the 
program only when the number of experiments in the design equals 1. That 
is, only when a simple simulation run using the scenario data values is done. 
When there is more than one experiment in a design, the results are given in 
data files amenable to input in a graphical statistical analysis package such as 
GRAFSTAT. 
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5.0 DETERMINING RELIABILITY GOALS 



In the previous sections, we have produced descriptions of the analytic 
reliability measure ef, and of the simulation analysis system. In the most 
fundamental terms, ef is a simple function of reliability measures of 
individual ammunition and gun system components. The simulation also 
uses the reliability data, as well as other system performance characteristics 
and an NGFS scenario as input. The output of the simulation is a set of battle 
goal results, as listed in Figure 4.3. Let us denote these k battle goal values by 
the vector bg(bgi, bg 2 , ..., bgk). 

The reason for developing the simulation model is that components of bg 
have relevance and meaning to our prospective decision maker. Given a 
vector of battle goal results, we expect the decision maker to be able to 
determine if the performance of the system is acceptable. Pursuing this 
expectation, we compel the decision maker to specify a function 

^ { 0 , 1 } 



so that T'(bg) = 1 if and only if the outcome bg is acceptable in terms of 
performance, 'F(bg) = 0 if not. For simplicity, we will assume that 'P has the 
following form: 




iffbgi<bgi i = l, 2, ...,k 
otherwise. 



(5.1) 



Hence bg* = 



» 






bgi,b&,..., bgk 



is the decision maker's threshold performance 
vector. Obviously some ^sign-switching will be necessary so that this 
threshold vector makes physical sense. 
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We propose that bg is a function, 

bg: [0,1] 

so that each battle goal is a function of ef, bg(e/) € 91'^. Hence, we subnait the 
following procedure for establishing reliability goals for major caliber 
ammunition components: 

1. specify a set of N experimental reliabilities for each ammunition 
component for each round type. 

2. specify bg*; 

3. i = l,2, ...,N 

i. Calculate e/i; 

ii. estimate bg(e/ i) via AMMO; 

4. ef* = min[efi: 'F(bg(c/, )) = l} 

The validity of the above procedure is based on the assumption that bgi is 
nondecreasing in ef for each i. 

Example 2. 

From our preliminary analysis, we chose the battle goals as follows: 
bgi = total mission time; 

bg 2 = total firepower of threat at 1 hour after engagement begins 
bg 3 = time-integrated point value of targets 

bg 4 = percentage of targets left undestroyed at time of invasion, 48 hours 
after engagement begins. 

Obviously the best of all possible situations is to have bg = [0,0,0,0]^. 

We expect the decision maker to specify the threshold vector bg* so that 
we can, in turn, provide a threshold ef*. Obviously, lower values of the 
components of bg* will face the value of ef* up. The usefulness of ef* is that 
it relates directly to gun and ammunition component failure probabilities. 
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5.1 ENHANCING EFFECTIVENESS 

Given that the decision maker has specified a threshold level of 
performance bg*, how must we deal with this threshold? For each round 
type, we are required to meet the attendant effectiveness threshold ef*. The 
round types are considered to operate in a fixed environment, including the 
reliability performance of the gun system the round is used with. Thus, of all 
of the system component reliabilities that impact the calculation of ef for a 
round type, only the round component reliabilities are under our control. 

Let us consider one round type with round components 1, 2, ..., k and 
exogenous components k + 1, k + 2, ..., N the exogenous components being 
those of the intended gun system. We hope to provide round to the fleet 
such that pi, p 2 , ..., pk- The k probabilities of failure for each of the round 
components are small enough so that ef* > ef*. If this is not the case, then 
one or more of the component reliabilities must be improved in order for the 
round to conform to the specified reliability threshold. 

The decision maker we allude to actually performs two roles, he procures 
ammunition components for assembly with currently held components to 
make new rounds, and he monitors the reliability of existing stockpiles of 
ammunition. When monitoring the stockpiles, he decides whether or not to 
perform some set of rework upon an ammunition lot to improve the 
reliability performance of the lot. 

5.1.1 The Procurement Decision 

For the procurement process, we will assume that the decision maker 
currently has in place procurement mechanisms for k-1 of the round 
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components, and has a high degree of confidence in the value of Xj, 
j = 1, 2, N, j i, the failure rates of the individual components. He needs a 
reliability goal for component i, that is, he needs to specify the maximum 
allowable xj which will produce a round conforming to the overall reliability 
threshold. 

From Section 2.3, we can state the reliability threshold constraint as 



ef* < 



X'r;(l-F;)/F;+1 



(5.2) 



Isolating pi, this constraint becomes 






where A is given by 



A = 



^ 1 " 






NT, 






Ti 



-1 



(5.3) 



(5.4) 



Tracing pi to its origins as a function of the failure rate x,, we arrive at the 
constraint 















(5.5) 



This is our reliability goal for component i. 

We have related the constraint of the effectiveness of a round to the 
failure rate of the component to be procured. We recommend that acceptance 
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tests for the new component establish Xj as the upper control limit for the 

percent defective in all procured lots of component i. 

Finally, if component i is to be used to build rounds of several different 

types, the reliability goal should be set by the round type establishing the 
* 

lowest Xj. 

5.1.2 The Surveillance Decision 

Ammunition surveillance is the practice of removing several rounds 
from a stockpiled lot of ammunition and testing the rounds. This testing is 
done in two modes, these being total-system and component-wise. Currently, 
the acceptance criteria for pronouncing a stockpiled lot fleet-ready are ad hoc. 
We propose replacing the current surveillance test criteria with those that 
will ensure that the lot tested exceeds ef* in efficiency. 

Usually, only a subset of the components in a round are selected for 
component-wise surveillance testing. From the tests, we can easily establish 
an upper confidence limit on the value of x, for each round component i 
which is tested. 

Components that cannot or are not tested component-wise are tested in 
the total-system test. During the total system test, the number of failures 
which is interruptive should be counted and classified, so that upper control 
limits are established for pi, the expected number of failures of type i which 
occur between successful rounds. Failures of types which are attributed to 
components tested component-wise are used to sharpen the estimates of the 
failure rates for these components. 

Thus, after the initial surveillance tests, we will have an estimate of ef for 
the lot. If this estimate is above ef*, we allow the lot to remain in the 
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stockpile and proclaim the lot to be conforming to our reliability goal. If the 
lot is not conforming, the decision maker has the option of replacing one or 
more of the components in each round in the lot. He has knowledge of the 
failure rate of the replacement components he has at hand, so he may 
determine whether performing the replacement will bring the lot into 
conformance with the reliability goal. No doubt, this rework decision has an 
economic aspect we do not address. 

5.1.3 Evaluating Marginal Improvements 

As we have developed ef as our performance measure, it is incumbent 
upon us to exploit the functional nature of ef to determine relationships 
between incremental improvements in reliability, training, or operations and 
fleet effectiveness in NGFS. We do this by taking partial derivatives. 



Consider the following two equations; 




9T,- Pi > 


(5.6) 


dp, Pi ^ 


(5.7) 



As 0 < Pi, cf < 1, we know that def/dTi e [-1,0], while def/dp\ e [0,1]. 

Equation 1 primarily addresses training of crew members, as better trained 
crews will shorten recovery times from gun or ammunition failures. 
Equation 1 tells us that the greatest operational payoff comes from reducing 
the recovery time of failures which occur most frequently, that is, with low 
values of pi. Note further that Equation 1 does not explicitly involve Ti, so 
that we see clearly that frequency of failure is more important than severity. 
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Equation 2 tells us that incremental improvements in reliability of 
individual components are ordered by the ratio T\/ pi- The greater this ratio, 
the more important an incremental improvement in Xi will be. Note that 
Tj/pi is large when Tj is large and . . . 

Xj is large. The upshot of this discussion is that 

i) crews should train with the goal of reducing the recovery time for the 
most frequent failures; 

ii) contractors, and procurement agents should weigh incremental 
improvements in component reliability using the ratio Tj/pi. 

Conclusion (ii) has direct impact on determining bonus values for contractors 

who exceed reliability specifications in their contracts. 

5.2 Output Analysis 

An experiment was designed whose purpose was to determine the 
functional relationship between each of the battle goals and the value of ef. 

The experiment consisted of three steps: 

i) the simulation for K of different combinations of round component 
failure rates, executing M replications per combination; 

ii) for each of the KM runs, exploit the difference between the observed 
value of ef and its known expectation to adjust the battle goal outcome 
for the run; 

iii) develop a set of regression metamodels used to determine the 
functional relationship between each battle goal's expected value and 
the theoretical value of ef. 

The experiment was performed for K = 34 and M = 200, using a stylization 
of a scenario provided by NWSC. The data used was perturbed from realistic 
values so as to avoid any classification constraints. In the following 

discussion, we will treat the following two battle goals, 

i) bgi = mean mission duration; 

ii) b^ = time average of enemy's firepower; 
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where firepower is given by each target's value, see Section 3.1.1. A direct 
relationship between the time average of the enemy's firepower and expected 
troop loss can be made if one (unrealistically) expects the amphibious assault 
to begin at the beginning of the scenario. 

Figures 5.1 and 5.2 show the outcome of our experiment. In each figure, 
each datum represents the average response from the 200 runs of one failure 
rate combination, after adjustment as described in Step ii. This adjustment 
was performed using the linear control variate method, see Bratly, Fox, and 
Schrage [1983]. 

Briefly, this method consists of modifying each response from the 
simulation as 

’^i = yi-PH~efl (5.8) 

where Yj is the i*^ replication of the random variable Y, and e/,- is the 
observation of the random variable EF. Since E[EF-e/j = 0, we have 
Ej^y] = E[Vf] for each i. it is well known that the best choice of j3 is given by 

cov[£F,y]-^^. (5.9) 

var[EFj 

as we don't know the value of /3, we estimate the required quantities from 
each of the K experimental combinations of failure rates. These estimates are 
shown graphically in Figure 5.3. Figure 5.4 graphically illustrates the 
reduction in variance in the mean response of each of the experimental 
combinations for the time-averaged fire power battle goal. The reductions in 
variance are in the neighborhood of 40-50% for this battle goal. Thus, by 
exploiting the difference between the simulation's experimental outcome for 
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ef and the true expected value, we have made our procedure 40-50% more 
efficient. 




Theoretical ef 

Figiure 5.1. Battle Goal 1 Plotted as a Function of Theoretical ef 
For the two battle goals, we see the fit of the two regression metamodels 
we constructed using standard techniques, see Kleijnen [1975]. Our estimates 

of the linear relationship between ef and each of the battle goals is given as 
bgi = 427.62 - 415.79 e/ (5.10) 

b^ = 1693.2 - 1069.4 ef (5.11) 

Both fits were subjected to standard regression diagnostics, and both were 
determined to be acceptable. 



2/22/91.11:38 AM.41 



41 




Theoretical ef 

Figure 5.2. Battle Goal 2 Plotted as a Function of Theoretical ef 
It is worth noting that with our combination of variance reduction technique 
and linear metamodel, we are regressing our battle goal outcomes on the 
quantities efi - ef and ef, and that these two calculations are pursued for very 
different ends. 

Setting Reliability Goals, an Example 

Suppose that the decision maker makes the determination that for the 
targets and scenario described, that the battle goal threshold is bg* = (100,860). 
That is, the mission execution is determined to be below acceptable levels of 
performance if it takes 100 minutes, or if the time-integrated fire power score 
is greater than 860. Using the regression metamodels, we see that executing 
the mission time constraint implies that ef > 0.788, while meeting the fire 
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power constraint implies that ef > 0.779. Thus, from this simple analysis, we 
can conclude that the threshold ef is given as 0.788. 




Figure 5.3. Estimated Correlation Coefficients between Time-integrated Fire 

Power and (ef- ef) 
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Figure 5.4. Variance Reduction for each Experimental Setting 
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6.0 CONCLUSION 



What we have endeavored to provide to the managers of the U.S. Navy's 

major caliber ammunition program are the following; 

i) a measure of performance which relates defect rates of ammunition 
components to battle force performance; 

ii) a simulation model to assist government analysts in establishing 
threshold values for the performance measure; 

iii) a methodology for prescribing maximum defect rates for ammunition 
components to ensure that new and stockpiled ammunition conforms 
to specified threshold levels. 

Hence, we have an important set of tools useful in managing procurement 
and surveillance of ammunition. 

It is our view that these tools should be integrated into a system of 
evaluation, physical experiment, data production, and procurement and 
surveillance decisions. The weakness of the tools we have produced thus far 
are that the simulation model is not verifiable — we cannot match simulation 
output with that of a physical experiment. Figure 6.1 shows our concept of an 
integrated reliability performance evaluation system. What follows are 
remarks concerning each component of this system. 

6.1 LIVE FIRE EXPERIMENTS 

Naval forces use their major caliber guns in exercises and qualification 
tests and equipment acceptance tests throughout the year. The gun 
component reliability and recovery time data available through current 
sources are not very complete and do not address the needs of performance 
analysis of ammunition. The establishment of our analysis tools should 
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compel the major caliber ammunition program to identify and collect data 
from these live fire experiments, especially with regard to gun system failure 
rates and recovery time. 



Assessment 



A. Live Fire 
Experiment 



Process 





D. Procurement/ 



Surveillance 

Decisions 




B. Produce 
Scenario 
Data from 
Outcome 



C. Simulate to 
Produce 
Operational 
Evaluation 




Figure 6.1. Integrated Reliability Performance Evaluation System 

6.2 PRODUCE SCENARIO DATA FROM OUTCOME 

Using the data from the live fire experiments, scenarios data files should 
be produced. See Section 3.1. 

6.3 SIMULATE TO PRODUCE OPERATIONAL EVALUATION 

The produced data files should be subjected to the same analysis process 
we have executed in Section 4 for the NSWC-supplied scenario. From this 
analysis, reliability thresholds are established for use in the management of 
procurement and surveillance processes. 
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6.4 PROCUREMENT/SURVEILLANCE DECISIONS 

Using the threshold values established, the management decisions 
required are made. This will result in ammunition getting to fleet user 
which has been affected by this process. The ammunition will thus be used in 
subsequent live fire experiments. Hence, there will exist a feedback from 
these decisions to the evaluation of ammunition. The process will become 
self-tuning. 
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